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I  Introduction 


A  general  model  for  free-response  data  has  been  proposed  by  the 
author  (Samejlma,  1972)  which  Includes  three  different  response  levels, 
l.e.,  the  nominal  response  level,  the  graded  response  level,  and  the 
dichotomous  response  level.  The  general  graded  response  model  is 
further  classified  Into  the  homogeneous  and  the  heterogeneous  cases. 

Two  of  the  most  common  uses  of  the  graded  response  model  may 
be  for  test  Items  which  have  more  than  two  Item  score  categories,  e.g., 

A,  B,  C,  D  and  F,  and  for  questionnaire  items  with  multiple  response 
categories,  e.g.,  “strongly  disagree,*'  "disagree,"  “agree,"  and  "strongly 
agree."  The  rationale  behind  this  general  model  Is,  however,  originally 
based  upon  the  general  sequential  cognitive  process,  which  has  been 
described  elsewhere  (Samejlma,  1967). 

Let  6  be  the  latent  trait  underlying  the  cognitive  process. 

When  the  latent  space  Is  unldlmenslonal,  6  Is  the  set  of  all  real  numbers, 
and,  when  It  Is  multidimensional,  6  Is  a  vector  of  k  sets  of  real 
numbers,  such  that 

(l.l)  » 

with  k  Indicating  the  dimensionality.  In  the  present  paper,  however, 
we  deal  solely  with  the  unldlmenslonal  latent  space. 

Let  us  take  problem  solving  as  an  example.  Suppose  that  for 
solving  the  problem  g  we  need  m  sequential  subprocesses.  Let  y 

O  ® 

denote  the  attainment  category  or  attainment  score.  One  must 


successfully  follow  all  the  m  sequential  subprocesses  In  order  to  solve 

O 

the  problem  g  ,  so  the  attainment  category  y  assumes  Integers,  0 

S 

through  (m  +1)  ,  with  y_  “  0  Indicating  that  the  Individual  subject  has 

O  O 

successfully  followed  no  subprocesses,  and  with  y  -  m  meaning  that 

o  o 

he  has  completed  all  m^  subprocesses  required  to  solve  the  problem.  The 
additional  attainment  score,  (m  +1)  ,  represents  that  the  subject  has 

O 

successfully  followed  the  additional  subprocess  which  Is  hypothesized  at 
the  end  of  the  entire  sequence  of  subprocesses.  Since  no  one  can 
accomplish  this,  the  conditional  probability,  given  6  ,  with  which  the 
subject  obtained  the  attainment  score  y  equals  zero,  regardless  of  the 

O 

fixed  value  of  .  With  this  setting,  we  can  see  that  the  general  graded 
response  model  can  readily  be  applied  to  the  single  strategy  case  of 
problem  solving.  In  this  paper,  however,  our  main  objective  is  to 
approach  a  general  model  for  the  multiple  strategy  case,  or  differential 
strategies.  In  the  context  of  latent  trait  theory. 

The  operating  characteristic  of  a  discrete  response  relates  the 
response  to  the  latent  trait  ^  .  In  the  context  of  problem  solving, 
this  operating  characteristic  Is  considered  with  respect  to  an  attainment 
score,  or  to  a  union  of  attainment  scores.  It  takes  an  important  role 
In  estimating  the  subject's  latent  trait  from  his  performance. 

As  concrete  mathematical  models,  we  shall  consider  those  In  the 
homogeneous  case,  which  was  expanded  and  generalized  from  the  one  on  the 
graded  response  level. 


It  Is  a  fairly  common  phenomenon  that  there  exist  more  than  one  way 


of  solving  a  problem.  In  proving  mathematical  theorems,  for  example,  we 
often  find  one  proof  plus  several  alternative  proofs  for  one  theorem.  To 
represent  those  differential  strategies  graphically.  Figure  2-1  presents  a 
simple  example  of  a  two  strategy  case  In  the  form  of  a  graph,  each 
strategy  having  a  small  number  of  subprocesses.  In  this  example.  If  we 
take  the  first  strategy,  then  we  must  traverse  the  path,  Vq  Vj^  V2  v^  v^  , 
whereas  we  must  follow  another  path  Uq  u^^  U2  u^  u^  u^  If  we  take  the 
second  strategy.  In  order  to  solve  the  problem.  Note  that,  although  we 

have  duplicate  symbols,  *  *^0  *  '^l  "  '*1  *  ^3  *  ^*4  ^^4  “  ^5  ’ 

These  two  paths  represent  the  two  successful  strategies  for  solving  our 
hypothetical  problem,  and  are  shown  as  two  directed  subgraphs  of  the 
original  graph,  or  digraphs.  In  Figure  2-2. 

When  the  subject  falters,  he  may  not  follow  either  of  the  two 
paths  shown  In  Figure  2-2.  In  such  a  case,  we  need  additional  arcs  In 
the  digraph  presented  as  Figure  2-1,  examples  of  which  are  shown  by  thin 
lines  In  Figure  2-3.  Two  examples  of  the  directed  subgraphs  of  Figure  2-3, 
which  represent  "faltering,"  are  presented  In  Figure  2-4.  In  the  first 
digraph,  the  subject  Initially  took  the  first  strategy,  but  came  back  from 

the  vertex  V2  to  the  vertex  v^^  ,  and  then  took  the  second  strategy  until 

he  solved  the  problem.  In  the  second  digraph,  the  subject  Initially  took 
the  second  strategy  up  to  the  vertex  u^  ,  then  came  back  to  the  vertex  Uj 
and  took  the  first  strategy.  These  are  rather  simple  examples  adding  one 

cycle  to  each  path  In  Figure  2-2,  making  the  strategy  a  trail  Instead  of  a 


f-ieUKt  Z-4 


Two  Examples  of  Directed  Subgraphs 
Representing  the  Subject's 
Fal tering. 


path.  We  can  conceive  of  more  complex  examples,  however,  In  which  the 
subject  traverses  several  cycles  repeatedly  In  a  single  walk,  for  Instance. 

So  far  In  our  graphs  all  the  vertices  are  those  Included  In  either 
the  first  or  the  second,  or  both,  of  the  two  directed  subgraphs  of  Figure 
2-2,  which  represent  the  two  successful  strategies  for  solving  the  problem. 
In  our  cognitive  process,  however,  we  often  choose  wrong  strategies  which 
do  not  lead  to  the  solution  of  the  problem.  Figure  2-5  Illustrates  such 
situations,  In  which  hollow  circles  are  added  to  the  solid  circles  of 
Figure  2-1,  and  dashed  arcs  are  added  to  Imply  additional  paths,  which 
represent  wrong  strategies.  Two  examples  of  such  unsuccessful  strategies 
are  Illustrated  In  Figure  2-6.  In  the  first  subgraph,  the  subject  followed 
the  second  successful  strategy  up  to  the  vertex  U2  »  and  then  branched 
himself  to  a  subsequence  of  two  subprocesses  which  led  him  to  a  "dead  end.** 
The  second  subgraph  Illustrates  a  similar  case  in  which  the  subject 
followed  the  second  successful  strategy  up  to  the  vertex  u^  ,  and  then 
followed  a  "dead  end”  subsequence  of  two  subprocesses. 

If  the  subject  took  a  wrong  strategy  and  found  out  he  could  not 
solve  the  problem  If  he  sticks  to  that  strategy,  he  may  give  up,  or  he  may 
come  back  to  a  previous  point  In  the  path  and  try  another  strategy.  To 
Illustrate  this  latter  situation,  additional  arcs  are  necessary  In  the 
graph  of  Figure  2-5.  In  Figure  2-7,  some  dotted  arcs  are  added  to  Imply 
new  cycles  which  the  subject  may  follow  when  he  "comes  back."  Two 
examples  of  such  trails  are  given  In  Figure  2-8.  In  the  first  example, 
the  trail  Indicates  that  the  subject  branched  himself  to  a  wrong  strategy 
after  he  reached  the  vertex  U2  of  the  second  successful  strategy. 


FIGURE  2-5 


Digraph  Added  by  Unsuccessful  Strategies 
(Dashed  Line). 


FIGURE  2-6 

Two  Examples  of  Unsuccessful  Strategies. 


FIGURE  2-7 


Digraph  Added  by  Loops  and  Cycles  Representing 
the  Subjects  Faltering  after  He  Has  Taken 
Unsuccessful  Strategies. 


Two  Trails  Representing  the  Subject's 
Faltering  when  He  Has  Taken 
Unsuccessful  Strategies. 


followed  the  path  until  he  reached  the  “dead  end,”  and  then  came  back  to 


U2  and  followed  the  rest  of  the  second  successful  strategy.  In  the 
second  example,  the  subject  followed  up  to  the  vertex  u^  of  the  second 
successful  strategy,  branched  himself  to  a  wrong  strategy,  followed  the 
path  to  the  "dead  end,”  and  came  back  to  u^  >  went  back  to  the  vertex 
Uj^  (“  Vj^)  ,  and  then  took  the  first  successful  strategy  until  he  solved 
the  problem. 

We  have  seen  Just  a  small  number  of  examples  of  conceptualizing  the 
cognitive  processes  Involved  In  problem  solving,  and  there  are  a  great 
many  other  varieties  of  paths,  trails,  and  walks  which  might  represent 
a  particular  subject's  cognitive  process.  The  subject  may  walk  the  same 
cycles  over  and  over  again,  for  example,  or  he  may  stop  at,  say,  the 
vertex  ^2  In  the  path  representing  the  first  successful  strategy  and 
then  not  proceed,  and  so  forth.  In  any  case,  graphic  representation 
Illustrated  In  this  section  will  benefit  us  In  systematizing  those  complex 
situations. 

Ill  Differential  Strategy  Tree 

As  was  referred  to  In  the  first  section,  latent  trait  theory  relates 
the  subject's  behavior  to  his  position  In  the  latent  space.  In  problem 
solving,  his  behavior  Is  observed  In  terms  of  the  degree  of  attainment 
toward  the  solution  of  the  problem. 

It  Is  obvious  that  following  those  cycles  presented  In  Figures  2-3 
and  2-7  Itself  will  not  directly  Improve  the  subject's  degree  of 
attainment  toward  the  solution  of  the  problem.  For  this  reason,  as  far  as 
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we  consider  the  degree  of  attainment  toward  the  solution,  we  can  more  or 
less  Ignore  the  subject's  traversing  on  cycles.  In  other  words,  the 
things  that  count  are  the  paths  In  those  graphs,  rather  than  trails  or 
walks  which  may  Include  one  or  more  cycles.  Thus  the  graph  of  Figure  2-7 
may  be  reduced  to  the  one  presented  In  Figure  3-1.  This  Implies  that, 
when  we  consider  the  subject's  degree  of  attainment  toward  the  solution, 
the  first  trails  In  Figures  2-4  and  2-8  are  treated  as  equivalent  to  the 
second  path  In  Figure  2-2,  and  the  second  trails  In  those  two  figures  are 
treated  as  equivalent  to  the  first  path  In  Figure  2-2. 

We  also  notice  that  even  the  same  subject  In  exactly  the  same 
problem  solving  situation  may  take  different  strategies.  If  It  Is  given 
repeatedly.  Thus  even  If  his  latent  trait  and  the  given  situation  stay 
the  same,  he  may  have  a  higher  probability  of  solving  the  problem  when  he 
chooses  one  strategy  In  preference  to  the  others.  In  other  words,  there 
Is  no  reason  to  assume  that  for  a  given  Individual  of  trait  6  the 
probability  of  success  stays  the  same  when  he  chooses  different 
successful  strategies.  In  the  Illustration  presented  In  the  preceding 
section,  the  probability  with  which  the  subject  of  trait  9  solves  the 
problem  using  the  first  successful  strategy  may  not  be  the  same  as  the  one 
when  he  uses  the  second  successful  strategy,  even  though  the  edge  (v^  v^) 
Is  the  same  as  the  edge  (u^  u^)  . 

The  above  observation  suggests  that,  as  far  as  we  consider  the 
degree  of  attainment  and  Its  corresponding  operating  characteristic,  it 
may  be  more  convenient  to  represent  the  set  of  all  strategies  as  a  tree. 


Figure  3-2  presents  such  a  tree  obtained  from  the  graph  In  Figure  3-1  for 


FIGURE  3-1 


FIGURE  3-2 

Differential  Strategy  Tree  Including  Both 
Successful  and  Unsuccessful  Strategies. 
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the  example  discussed  In  the  preceding  section.  Note  that  not  only  are 
the  two  points  V4  and  U3  In  Figure  3-2  the  same  point  In  the  original 
graph,  Figure  3-1,  but  also  the  two  hollow  circles  marked  with  *  in 
Figure  3-2  are  a  single  point  In  Figure  3-1,  and  so  are  the  two  marked 
with  **  .  Hereafter,  we  shall  call  this  kind  of  tree  the  differential 
strategy  tree.  It  Is  a  kind  of  directed  graph  which  contains  several  paths 
representing  different  strategies.  Joining  a  common  Initial  endpoint  with 
the  other  distinct  endpoints.  We  may  call  this  Initial  point  a  nothing 
point.  Indicating  that,  "nothing  has  been  accomplished  yet."  We  may  call 
the  other  endpoints  for  the  successful  strategies  solution  points, 
meaning,  that  the  "solution  has  been  reached."  Since  no  one  can  surpass 
a  solution  point.  It  also  represents  a  hypothesized  attainment  score 
which  no  one  can  obtain. 

In  theory,  we  should  be  able  to  estimate  the  operating  characteristic 
for  each  of  the  subprocesses  In  Figure  3-2,  which  are  represented  by  solid 
or  dashed  edges.  If  appropriate  models  are  discovered.  In  practice, 
however,  the  development  of  such  a  model,  or  models,  and  Its  validation 
will  Involve  much  more  complicated  problems.  If,  at  the  Initial  stage,  we 
Include  unsuccessful  strategies  In  addition  to  the  successful  strategies. 

For  this  reason,  here  we  shall  consider  models  for  successful  strategies 
only,  and  solely  use  the  Information  related  to  the  successful  strategies. 
When  some  model  thus  constructed  has  been  validated,  we  can  proceed  to 
explore  the  operating  characteristics  of  the  various  subprocesses  of 
unsuccessful  strategies.  In  a  way  similar  to  that  of  estimating  the 
plausibility  function  of  each  dlstractor  of  the  multiple-choice  test  Item 
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after  we  have  estimated  the  operating  characteristic  of  the  correct  answer 
(cf.  Samejlma,  ONR  Final  Report,  1981).  Thus  the  path  representing  the 
first  unsuccessful  strategy  in  Figure  3-2  is  treated  here  as  equivalent  to 
the  path  Vq  v^  V2  In  the  first  successful  strategy,  those  representing 
the  second  and  third  unsuccessful  strategies  to  the  path  v^  v^  In  the 
first  successful  strategy,  the  path  for  the  fourth  unsuccessful  strategy 
to  the  path  Uq  u^  U2  u^  In  the  second  successful  strategy,  and  the  one 
for  the  fifth  to  the  path  Uq  Uj^  U2  In  the  second  successful  strategy. 

The  example  we  have  observed  is  simple  and  easy.  In  reality, 
however,  we  should  expect  more  complicated  differential  strategy  trees. 
Figures  3-3  and  3-4  present  another,  a  little  more  complicated  example  of 
the  digraph  and  corresponding  differential  strategy  tree.  In  which  only 
successful  strategies  are  drawn.  Thus  In  our  second  example,  we  have  five 
successful  strategies  and  five  solution  points. 

IV  A  General  Model  for  Differential  Strategies 

Let  w  denote  the  number  of  successful  strategies  for  solving  the 
problem  g  .  This  number  equals  the  number  of  solution  points  In  the 
differential  strategy  tree,  which  was  illustrated  in  the  preceding  section. 
Each  of  those  w  strategies  consists  of  m^^^  (1  ■  l,2,...,w)  subprocesses, 

and  they  are  represented  by  the  vertices,  excluding  the  first  and  last. 

In  the  digraphs  and  In  the  differential  strategy  trees  presented  In  the 
previous  two  sections.  In  the  example  which  was  first  presented  and 
discussed  In  Section  2,  w  *  2  ,  mg^  ■  3  and  mg2  ■  4  ,  and  the 
subprocesses  are  represented  by  four  edges,  (vq  v^),  (v^  V2),  (V2  Vj) 


FIGURE  3-3 


Another  Example  of  Digraph  Representing 
Five  Successful  Strategies. 


^2  S3  55  -6 

(t2)  (t3) 


V 
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and  (Vj  v^),  in  the  path  Vq  V2  Vj  representing  the  first 
successful  strategy,  and  by  five  edges,  (uq  u^),  (uj  U2),  (u2  U3),  (uj  u^) 
and  (u^  path  Uq  Uj^  U2  u^  u^  u^  which  represents  the  second 

successful  strategy.  Hereafter,  we  shall  call  this  example  Example  1. 

It  Is  fairly  common  that  In  the  original  digraph  some  subprocesses  are 
shared  by  two  or  more  different  strategies,  as  we  can  see  In  Example  1, 
l.e.,  (Vq  Vj)  -  (uq  Uj)  and  (V3  v^)  -  (u^  Uj)  . 

Let  be  the  attainment  score  indicating  the  degree  of 

attainment  of  the  subject's  performance  toward  the  solution  of  the  problem 
g  ,  which  takes  on  Integers  0  through  m^^  when  the  subject  chooses 
the  strategy  1  .  Figure  4-1  presents  several  attainment  scores  assigned 
to  trails  which  were  taken  from  Example  1.  Also  Figure  4-2  shows  the 
attainment  scores  assigned  to  each  edge  of  the  differential  strategy  tree 
of  the  other  example,  which  was  presented  at  the  end  of  Section  3. 
Hereafter,  we  shall  call  this  other  example  Example  2. 

A  general  model  for  differential  strategies  concerns  the  assignment 
of  an  operating  characteristic  to  each  attainment  score  y^^  of  each  of 
the  w  strategies  1  for  solving  the  problem  g  .  By  such  an  operating 
characteristic  we  mean  the  conditional  probability  with  which  the  subject 
of  trait  6  chooses  the  strategy  1  and  obtains  the  attainment  score 
Fgi  •  We  notice,  however,  that  In  general.  If  the  subject's  performance 
stopped  before  branching,  there  Is  no  way  to  decide  which  of  the  two  or 
more  strategies  he  would  have  taken.  This  fact  is  well  represented  In  the 
differential  strategy  tree.  For  example.  In  Figure  3-4,  (sq  s^^), 

(^0  ^l^*  (“0  “1^*  ^*0  ''l^  *  single  edge  before 


7V V  /  r  rii  r~. 


■  0  or 


yg2  '  ° 


»o 


'3  »4 


^gi 


FIGURE  4-1 

Examples  of  Attainment  Scores  Given  to  Different 
Subgraphs  In  Example  1. 


strategy 

1 


attainment 

score 

ygi 


1  2 


FIGURE  4-2 

Attainment  Scores  Assigned  to  Each  Line  of 
the  Differential  Strategy  Tree  In 
Example  2. 
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the  first  branching,  and  as  long  as  the  subject's  performance  stays  with 
this  subprocess  he  could  be  heading  for  any  one  of  the  five  successful 
strategies.  We  can  also  say  the  same  for  the  edge  which  Is  both  (s^ 
and  (tj^  t2)  ,  for  the  one  which  Is  both  V2)  and  (w^  W2)  ,  and  so 

forth.  Thus  we  must  assign  a  single  operating  characteristic  for  each 
edge  of  the  differential  strategy  tree.  Since  each  edge  represents  a  union 
of  one  or  more  attainment  scores,  the  operating  characteristic  Is  to  be 
assigned  to  each  union.  For  Instance,  following  an  appropriate  model, 
a  single  operating  characteristic  will  be  assigned  to  the  union  of  y^^i^  "  0 
for  1  >  1,2, 3, 4, 5  ,  and  the  same  model  will  provide  us  with  an  operating 
characteristic  solely  for  yg^  “  3  ,  and  so  on.  For  convenience,  we  shall 
choose  the  smallest  1  In  each  union,  and  let  Fggi  denote  such  a  union 
with  s  for  the  actual  attainment  score.  In  Example  2,  for  Instance, 

y|01  “  <ygl“°>  ^  ^  ^yg3"°>  ^  ^  5^834  "  ’ 

and  none  of  the  four  unions,  yg02  *  ^03  *  ^g04  ^8^5  *  ®*l-8ts. 

Let  IL*  (6)  denote  the  conditional  probability  with  which  the 

subject  of  trait  6  obtains  s  as  his  attainment  score  In  one  of  the 

strate8les  which  belongs  to  yg3i  »  with  the  joint  condition  that  he  has 

already  obtained  the  score  s-1  .  Since  there  Is  no  preceding  attainment 

score  for  y^j^  -  0  ,  and  ygoi  is  the  union  of  ygj  “  0  for  all  the  w 

strategies,  the  attainment  function  M  ^  (0)  takes  on  unity  throughout 

ygOl 

the  whole  range  of  0  .  On  the  other  hand,  since 

hypothesized  attainment  score  which  Is  higher  than  the  full  score  mg^  , 
the  attainment  function  (9)  assumes  zero  for  the  entire  range 

^g(mgi+l)l 

of  0  for  each  of  the  w  strategies.  Thus  we  can  write 


8-0 


-  ,  8  -  nigi+1 


Note  that  in  (4.1)  the  first  line  Indicates  a  single  function  for  the 

union  of  -  0  for  1  -  l,2»...,w  ,  while  the  second  line  Indicates 

w  separate  functions  for  1  -  1,2 . .  . 

Hereafter,  we  shall  assume  that  each  attainment  function  (6) 

^gsl 

Is  three-times  differentiable  with  respect  to  6  .  Note  that  this 

assumption  does  not  contradict  (4.1). 

Let  P*A  (6)  be  the  conditional  probability  assigned  to  the  union 

ygsi 

of  attainment  scores  ygg^  >  with  which  the  subject  of  trait  6  chooses 
a  strategy  which  belongs  to  ygg^  and  obtains  the  attainment  score  s 
or  greater.  We  shall  call  this  function  the  cumulative  operating 
characteristic  of  the  attainment  score  union  y*„.  . 


From  the  definitions  of  this  function  and  the  attainment  function 
C0)  >  we  can  write 

"gsl 


(4.2) 


PS*  (0) 

^gsl 


k-0 


where  *1  Indicates  the  closest  Integer  less  than  or  equal  to  1  for 
which  the  union  of  attainment  scores  exists.  In  particular,  we  have 


(4.3) 


8-0 


8-1 


8  -  mgi+1 


Note  that  the  first  line  of  (4.3)  Indicates  a  single  function  for  the 
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unlon  of  ■  0  for  1  »  ,  the  second  line  Indicates  one  or 

more  functions  depending  upon  the  branching,  and  the  third  line  Indicates 

the  w  separate  functions  for  1  -  l,2,«..,w  .  In  Example  1,  the  second 

line  Includes  two  functions,  while  In  Example  2  It  Includes  three  functions. 

Let  A**  (0)  be  the  first  derivative  or  the  natural  logarithm 

"gsl 

of  the  cumulative  operating  characteristic  P**  (0)  ,  that  is, 

"gsl 


Note  that  this  function  is  unchanged  when  (e)  la  multiplied  by 

ygsl 

a  constant.  To  be  more  specific,  let  'i'(0)  be  a  function  defined  by 


(4.5) 


where  0  £  c^  <  C2  £  1 

Pv*  (®)  given  by 
^gsl 


.  The  first  derivative  of  the  natural  logarithm  of 
~log[ci  +  (C2-Ci)l'(0)]  ,  which  equals  -^logfCe) 


If,  and  only  If,  c^  >  0  .  The  formula  (4.5)  has  been  observed  In  a 


somewhat  different  context  (Samejlma,  ONR/RR-82-1)  where  P**  (0)  Is 

ygsl 


replaced  by  any  strictly  Increasing  function  of  @  with  zero  and  unity 


as  Its  two  asymptotes.  We  have  called  the  four  different  types  of  functions 


derived  from  (4.5)  Types  A,  B,  C  and  D,  depending  upon  the  values  of  c^^ 
and  C2  I  i.e.,  the  function  Is  of  Type  A  when  0  -  Cj^  <  C2  ■  1  ;  of  Type  B 

when  0  <  Cj  <  C2  “  1  ;  of  Type  C  when  0  ■  Cj^  <  C2  <  1  ;  and  of  Type  D 

when  0  <  Cj^  <  C2  <  1  »  respectively.  This  Implies  that  the  cumulative 

operating  characteristic  of  Types  A  and  C  may  share  the  same  function  for 
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^  same  for  those  of  Types  B  and  D.  The 

ygsl 

necessary  and  sufficient  condition  that  M  *  (0)  be  strictly  increasing 

ygsi 

in  0  is  that  the  inequality 


(4.6)  A**  (0)  >  A*  *  8  -  1.2 . (m_i+l) 

ygsl  ^yg(s-i)(*i)i  8^ 

holds  almost  everywhere  with  respect  to  0  . 

The  operating  characteristic,  P  *  (0)  ,  defined  for  the  union 

ygsl 

of  the  attainment  scores  l8  given  by 


(4.7) 


.(6) 


ygsl 


P* 

ygsl 


*  .(e)  - 


I 

j* 


p*- 

yi(8+i)j 


*.  ...  .(e) 


where  Z  indicates  the  summation  over  all  the  strategies  j  branching 

j* 

from  the  point  which  lies  immediately  after  the  line  representing  yggi  • 
This  operating  characteristic  can  be  considered  as  the  likelihood 
function  in  estimating  the  subject's  latent  trait  0  .  If  there  are  more 
than  one  problem  satisfying  the  conditional  Independence  of  the  attainment 
scores  across  the  different  strategies,  given  0  ,  the  maximum  likelihood 
estimation  of  the  subject's  latent  trait  can  be  performed  on  the  basis  of 
the  response  pattern  V  ,  such  that 


(4.8)  V  -  (yii^,  y2i2.  •••  .  ygi^.  •••  . 

for  the  n  problem  solving  tasks,  where  1^  is  a  strategy  for  solving 
the  problem  g  and  yg^  is  the  attainment  score  when  the  subject  chooses 


the  strategy  i  for  solving  the  problem  g  .  Let  P  (0)  be  the 
8  V 

operating  characteristic  of  the  specific  response  pattern  V  .  We  can  write 


(4.9)  P  (0)  =  n  P  .  (6)  . 

v  *  “gsl 


where  K  Indicates  the  multiplication  over  every  union  yggi  which 
an  element  of  V  belongs. 

It  is  beneficial  to  search  for  a  family  of  models  which  provide  us 
with  a  unique  maximum  for  every  possible  response  pattern  given  by  (4.8). 
This  can  be  done  as  a  generalization  of  the  unique  maximum  condition 
proposed  for  the  graded  response  model  (cf.  Samejima,  1969,  1972).  Two 
of  such  models  will  be  proposed  and  observed  in  Section  5. 

The  basic  function,  A  *  (0)  ,  for  the  union  of  attainment  scores 

- y|si 

y*gj^  is  defined  by 


The  maximum  likelihood  estimate,  0y ,  of  the  subject's  latent  trait  based 
upon  his  response  pattern  is  given  as  the  solution  of  the  likelihood 
equation  such  that 


(4.11) 


log  P  .  (0) 


1  A  *  (e)  , 

*  ^gsl 


where  I!  Indicates  the  summation  over  every  union  y*  4 


to  which  an 
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element  of  V  belongs.  A  sufficient  condition  that  a  unique  modal  point 

exists  for  the  likelihood  function  Py(6)  of  each  and  every  response 

pattern  V  Is  that  this  basic  function  Is  strictly  decreasing  In  6  with 

non-negative  and  non-posltlve  values  as  Its  two  asymptotes,  for  every 

union  Ygai  •  This  can  be  shown  In  the  same  way  that  we  did  for  the  basic 

function  A  (6)  of  the  graded  Item  score  (cf.  Samejlma,  1969)  . 

*g  * 

For  brevity,  sometimes  we  call  this  condition  the  unique  maximum  condition. 

Multl-correct  and  multl-lncorrect  responses  have  been  discussed 
briefly  (Samejlma,  1972,  Sections  3.4  and  4.2),  and  It  has  been  pointed 
out  that  all  those  response  categories  can  be  syndrome  categories, 
l.e.,  response  categories  whose  operating  characteristics  satisfy  the 
unique  maximum  condition  (Samejlma,  1969,  1972).  Those  cases  are  Included 
In  Situation  B  (Samejlma,  1972,  Section  3.2),  where  there  exist  more  than 
one  response  category  whose  operating  charaterlstlc  Is  of  Type  (1)  (or  of 
Type  (11)),  and,  at  least,  one  response  category  whose  operating  characteristic 
is  of  Type  (11)  (or  of  Type  (1)),  and,  in  addition  there  may  be  one  or  more 
response  categories  whose  operating  characteristics  are  of  Type  (111).  It 
should  be  recalled  that  by  Types  (1),  (11)  and  (ill)  we  mean  strictly 
Increasing,  strictly  decreasing,  and  unlmodal  operating  characteristics 
which  satisfy  the  unique  maximum  condition.  In  Situation  B,  the  sum  total 
of  the  upper  as3nnptotes  of  all  the  operating  characteristics  of  Type  (1) 
should  equal  unity.  This  Implies  that,  as  6  tends  to  positive  Infinity, 
the  operating  characteristic  of  each  correct  answer  approaches  a  finite 
positive  value  less  than  unity.  We  notice  that  such  operating  characteristics 
belong  to  the  Type  C  model  (Samejlma,  ONR/RR-82-1). 
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Slmllarltles  between  the  differential  strategies  In  problem  solving 
and  the  multi-correct  responses  In  testing  are  obvious.  If  we  consider 
two  or  more  different  strategies  which  lead  to  the  solution  of  the  problem 
as  two  or  more  different  answers  to  a  question,  then  they  will  be  treated 
as  multi-correct  responses.  We  can  see  that  the  concept  of  multi- 
correct  responses  can  be  transferred  to  differential  strategies,  when  there 
exist  more  than  one  successful  strategy  In  solving  a  problem. 

V  Homogeneous  Case 

The  Homogeneous  case  of  the  graded  response  level  has  been  developed 
and  discussed  (Samejlma,  1972)  as  a  generalization  of  a  family  of  models 
on  the  dichotomous  response  level.  Sufficient  conditions  that  a  model 
provides  us  with  a  unique  modal  point  for  the  likelihood  function  of  each 
and  every  response  pattern  have  been  investigated.  In  the  homogeneous 
case,  a  sufficient  condition  Is  that,  for  an  arbitrary  Item  score  Xg 

(jt  0)  ,  the  cumulative  operating  characteristic  P*  (0)  Is  of  Type  A, 

g 

l.e.,  strictly  Increasing  In  0  with  zero  and  unity  as  Its  two  asymptotes, 
and  its  asymptotic  basic  function,  A  (9)  ,  which  Is  defined  by 

Is  Strictly  decreasing  in  0  .  The  satisfaction  of  this  sufficient 
condition  also  implies  two  desirable  features  of  the  model  such  that:  1) 
the  operating  characteristic  of  each  graded  Item  score  of  each  Item  has  a 
single  modal  point,  and  2)  those  modal  points  for  a  single  Item  are 


-24- 


arranged  In  the  same  order  as  the  item  score  itself.  The  normal  ogive 

and  logistic  models,  which  have  been  generalized  from  the  corresponding 

models  on  the  dichotomous  response  level,  are  two  examples  of  the  models 

which  satisfy  the  above  sufficient  condition. 

These  models  of  the  homogeneous  case  on  the  graded  response  level 

can  be  generalized  to  provide  us  with  those  which  belong  to  the  general 

model  of  differential  strategies.  Let  'i'(Q)  be  a  function  of  Type  A. 

We  shall  consider  the  cumulative  operating  characteristic,  (6)  t 

ygsl 

of  the  union  of  attainment  categories  Tggi  such  that 


(5.2) 


«y|.i 


where  ®  *  is  negative  infinity,  o  is  positive  infinity 

ygoi  yg(m  +1)1 

for  1  ■  l,2,...,w  ,  and  the  values  of  Uy*  ^  are  ordered  in  the  same 

way  as  those  of  s  ,  for  every  strategy,  and  S,.*  is  a  constant 

^gsl 

which  satisfies 


(5.3)  I  6y*  ■  ®y*/  ’ 

j*  ^gsj  ^g(s-l)l 

with  I  indicating  the  summation  over  all  the  strategies  J  branching 

j* 

from  the  point  of  the  dlfferentall  strategy  tree  which  is  located  right 

after  the  line  representing  the  union  yg(g_i)i  •  Ffo®  (5.3)  it  is 

obvious  that,  as  far  as  there  is  no  branching,  ^  ^  « 

ygsl  yg(s-i)i 

A  sufficient  condition  that  the  model  satisfies  the  unique  maximum 
condition  is:  1)  that  the  values  of  the  constant 


are  the  same  for 
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all  the  strategies  j  which  branch  from  the  vertex  located  Immediately  after 
the  edge  representing  ygsi  «  2)  that  we  have 


(5.4)  <  0 


almost  everywhere  In  the  domain  of  6  .  To  prove  this,  we  obtain  from 

(4.7),  (5.2),  (5.3)  and  the  definition  of  the  basic  function  A  *  (9)  , 

ygsl 

which  was  given  by  (4.10), 


(5.5) 


log  Py.^^(9) 


<f'(e-ci^*  ) 

ygsl 


®>'g<.+l)l 


where  ^  Indicates  the  summation  over  all  the  strategies  j  branching 

J* 

from  the  vertex  which  lies  Immediately  after  the  line  representing  the 
union  ygsl  •  ®y  of  the  first  condition,  we  can  rewrite  (5.5)  In 

the  form 


(5.6)  *y.^^(e)  -  ^  log 


log  {^(e-a  *  )  -  ne-ct  * 

30  ygsl  yg(B+l)j 


)}  . 


We  notice  that.  If  we  replace  ygsl  ty  the  graded  Item  score  Xg  and 

use  ’*'(9-0  )  as  the  cumulative  operating  characteristic  P*  (0)  ,  the 

g  8 

last  form  of  (5.6)  Is  Identical  with  the  basic  function  of  the  graded  Item 
score,  and  the  left  hand  side  of  (5.4)  Is  Identical  with  the  corresponding 
asymptotic  basic  function.  Thus  we  can  say  that  all  the  unions,  ygsl  > 
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are  equivalent  to  syndrome  response  categories  (cf.  Samejlma,  1972, 

Section  5.2),  and  a  unique  maximum  Is  assured  for  every  possible  response 
pattern . 

If,  for  example,  '(^6)  Is  a  normal  ogive  function  or  a  logistic 
distribution  function,  then  (5.4)  Is  satisfied  (Samejlma,  1972,  Section 
5.2),  and  we  can  develop  the  normal  ogive  model  and  the  logistic  model  In 
the  context  of  the  general  model  for  differential  strategies,  and  both 
of  them  satisfy  the  unique  maximum  condition.  In  these  two  models,  the 
cumulative  operating  characteristics  are  defined  by 

(5.7) 

and 

respectively,  where  a  ( >  0)  Is  the  discrimination  parameter  specific 

O 

for  each  problem  g  ,  b„*  Is  a  difficulty  parameter  defined  for  each 

^gsl 

union  of  attainment  scores,  with  b„*  “  and  b  *  ■  *  and 

ygOl  ^g(>%l  +l)i 

all  those  values  are  arranged  In  the  same  order  as  s  with  respect  to 
each  strategy,  and  D  In  (5.8)  Is  a  scaling  factor  which  assumes  1.7  to 
retain  the  same  set  of  parameter  values  as  those  In  the  normal  ogive 
model . 

For  the  purpose  of  Illustration,  we  shall  consider  a  differential 
strategy  tree  shown  In  Figure  5-1,  which  was  taken  from  Example  1  given  In 
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Sectlon  1  and  has  only  two  successful  strategies,  or  w  «  2  .  Note, 

however,  that  this  tree  may  represent  either  one  of  the  digraphs  shown  In 

Figure  5-2.  Figure  5-3  presents  the  set  of  ten  cumulative  operating 

characteristics  PA*  (6)  ,  with  a 

^gsl 

'’y*  ”  -1.00,  '’y*,  “  0.50,  by*  “  -1.80  ,  b  *  ■  0.00  and 

yg2i  yg3i  yg22  yg32 

b  *  ■  2.00  ,  S  ^  ■  0.60  and  g,,*  “  0.40  .  The  cumulative 

yg43  ygii  ygi2 

operating  characteristic  for  YgOi  drawn  by  a  solid  line,  and  the 
three  cumulative  operating  characteristics  for  the  first  strategy  are 
drawn  by  dotted  lines,  the  four  for  the  second  strategy  by  dashed  lines, 
and  the  two  for  Fgsi  yg62  ‘Coincide  with  the  abscissa.  Figure  5-4 

presents  the  corresponding  operating  characteristics.  In  which  all  the 
modal  points  except  for  the  negative  and  positive  Infinities  are  shown. 

VI  Single  Strategy  Case 

Suppose  that  In  solving  the  problem  g  there  Is  only  one  successful 
strategy.  Such  a  single  strategy  case  can  be  considered  as  a  special  case 
of  the  multiple  strategy  case  In  which  w  -  1  .  The  union  of  attainment 
scores,  therefore,  Includes  one  and  only  one  attainment  score,  and  we  can 
write 

ygsi  ”  (yg  "  ' 

where  y  Is  the  attainment  score  of  the  unique  successful  strategy. 

8 

Thus  the  three  functions,  l.e.,  the  attainment  function,  the  cumulative 
operating  characteristic  and  the  operating  characteristic,  are  defined 
for  each  attainment  score,  rather  than  the  union  of  attainment  scores. 


FIGURE  5-1 


Differential  Strategy  Tree  of  Two 
Strategies  Taken  from  Example  1. 


and  (y  2  =  4),  Respectively  (Dashed  Line). 


Let 


Mv  (0)  f  PJ  (0)  and  P  (0)  denote  these  three  functions  for  the 

70  Xo  7o 


'g  -^g  ■'g 

single  strategy  case.  We  can  change  the  formulas  (4.1),  (4.2),  (4.3), 


(4.7),  (4.8)  and  (4.9)  Into  much  simplified  forms,  such  that 


(6.2)  IT  (0)  I 

^g  I  -  0 


(6.3) 


(6.4) 


(6.6) 


and 


(6.7) 


y  “  » 

•^g  g 


g 


P*  (6)  .  n“  M  (0)  y  -  0,l,...,m  ,(m  +1)  , 

Xg  J  ®  g  8 


f-  1 


P*  (0)  < 


'g 


y  -  0 


(0)  y„  -  1 


I  -  0 


Xg  -  Og+1  . 


(6.5)  P  (0)  -  P*  (0)  -  P^  .i)(0) 


Xg  “  0,l,...,mg  , 


“  <yi'  . . yg . V 


Pv(9)  -  n  Py  (0)  . 

y  EV  8 
•^g 


where  m  Indicates  the  number  of  the  subprocesses  Involved  In 
g 


solving  the  problem  g  . 

We  notice  that  these  results  are  Identical  with  those  formulas  for 
the  graded  response  model  (Samejlma,  1972),  with  the  replacement  of  the 


Item  score  x  by  the  attainment  score  y  .  Many  findings  concerning 

O  O 


» 
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the  graded  response  model,  therefore,  can  directly  be  applied  for  the 
single  strategy  case. 

Let  A*  (@)  be  the  first  partial  derivative  of  the  natural 

logarithm  of  P*  (0)  ,  such  that 
^g 

(6.8)  A*  (0)  =  log  P*  (0)  yg-0.1 . (®g+l)  • 

g  g 

It  has  been  shown  that  the  necessary  and  sufficient  condition  that 

M  (0)  be  strictly  Increasing  In  0  Is  that  the  Inequality 
^g 

(6.9)  A*  (6)  >  A^y  _^j(0)  yg  -  1 ,2 , . . . , (m^+l) 

holds  almost  everywhere  with  respect  to  0  (cf.  Same jlma,  1967,  1972) 
In  the  homogeneous  case  P*  (0)  has  zero  and  unity  as  Its  two 
asymptotes  for  y  »  1,2,...,  m^  ,  and,  furthermore,  we  can  write 

O  O 

(6.10)  P*(0)  -  P*(0  -  a^g)  , 

where  r  and  s  are  two  arbitrarily  selected  attainment  categories 
out  of  1  through  m^  with  r  <  s  ,  and  Is  a  positive 

finite  constant.  We  obtain  from  (6.8)  and  (6.10) 

(6.11)  A*(0)  -  A*(0  -  arg)  . 

From  (6.9)  and  (6.11)  It  Is  obvious  that  a  sufficient,  though  not 
necessary,  condition  that  M  (0)  be  strictly  Increasing  In  0  for 


y  “  1,2, ...,m  is  that  A*  (0)  is  strictly  decreasing  In  6  for 

8  g  yg 

an  arbitrarily  chosen  attainment  category  out  of  1  through  m  .  When 

O 

tends  to  positive  infinity  and  for  two  adjacent  real  attainment 

categories  tends  to  zero,  this  condition  tends  to  the  necessary  and 
sufficient  condition,  for  It  requires  that 


(6.12)  A*  (6)  >  A*  (0+e) 

g  g 


for  any  small  positive  value  of  0  .  Note  that  this  condition  Is 

satisfied  whenever  the  unique  maximum  condition  Is  satisfied.  Above  all, 

when  the  asymptotic  basic  function.  A*  (0)  ,  which  is  defined  by 

^g 


(6.13)  X*  (0)  -  [-^P*  (0)][  3  P*  (0)] 

30  30  ”g 


is  strictly  decreasing  In  0  ,  not  only  M  (0)  of  each  subprocess 
is  strictly  Increasing  in  0  ,  however  finely  differentiated  It  may 
be,  but  also  a  unique  maximum  Is  assured  for  the  likelihood  function 
of  each  and  every  possible  response  pattern  which  consists  of  such  attainment 
scores  of  different  tasks  (cf.  Samejlma,  1972,  Sections  5.1  and  5.2). 

It  has  been  shown  (Samejlma,  1967,  1972)  that  In  the  normal  ogive  model  and 
In  the  logistic  model  on  the  graded  response  level  (Samejlma,  1969),  for 
example,  this  condition  Is  satisfied.  In  the  former  example,  we  can 
write 


(6.1A) 


P*  (0)  -  (2 


-1/2  ) 
ir)  I  g  exp(-u^/2)  du 
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and  In  the  latter 


(6.15) 


P*  (6) 


[1  +  exp{-Da„(e-b„  )}] 


g 


'g 


where  a  is  the  Item  discrimination  parameter  and  is  the 

difficulty  parameter  for  each  attainment  category  y  ,  and  D  is  a 

O 

scaling  factor  which  is  usually  set  equal  to  1.7  (Blrnbaum,  1968). 
In  both  models,  the  upper  asymptote  of  M  (8)  for  y_  *  l,2,...,m 

Xg  g  g 

is  unity,  while  the  lower  asymptote  is  zero  in  the  normal  ogive  model 

and  exp[-Dag(by  -  b^y  _]^j)]  in  the  logistic  model.  This  lower 
g  g 

asymptote  in  the  logistic  model  depends  upon  the  distance  between  the 
difficulty  parameters  of  the  two  adjacent  attainment  categories. 


assuming  zero  for  ^  positive  numbers  less  than  unity 

otherwise.  In  both  models,  (ft)  for  y_  ■  2,3,...,m.  tends  to 

Xg  g  g 

unity  for  the  entire  range  of  0  as  b  approaches  b^  >  and 

tends  to  P*  (ft)  as  by  departs  from  b^y  (cf.  Samejima, 

g  g  g 


1972,  Figure  5-2-1). 

We  shall  consider  an  example  of  the  problem  solving  task,  in  which 
we  can  assume  a  single  successful  strategy.  The  problem  was  selected  from 
the  thirty  paper-and-pencll  test  items  of  the  LIS  Measurement  Scale  for 
Non-Verbal  Reasoning  Ability  (Indow  and  Samejima,  1962,  1966).  The  test 
item  is  called  ^2.2  ’  shown  in  Figure  6-1.  There  are  three 

Japanese  words  written  by  English  alphabetical  letters,  "YAMA,"  "UMI,” 
and  "KAWA,”  which  consist  of  one,  and  only  one,  of  the  seven  one-digit 
numbers,  which  are  shown  in  the  brackets  in  Figure  6-1.  The  sum  of  the 
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first  two  words  is  supposed  to  equal  the  third  word,  and  the  examinee  is 
expected  to  find  out  which  letter  represents  which  number. 

In  solving  this  problem,  we  notice  that;  1)  "I”  is  added  to  "A" 

to  produce  "A"  itself  with  no  carried  number  added,  so  "I”  must  equal 
0  ;  2)  a  similar  result  is  observed  for  "A”  and  "U"  ,  but  0  is 
already  assigned  to  "I"  ,  so  “U"  must  be  9  ;  3)  "K”  must  be  greater 
than  "Y”  by  one,  so  (Y,K)  must  be  either  one  of  (3,4)  ,  (6,7)  and 
(7,8)  ;  4)  since  1  must  be  carried  from  the  second  lowest  digit  to  the 

third,  two  times  ”M"  must  be  "W”  plus  ten,  so  (M,W)  must  be  either 
(7,4)  or  (8,6)  ;  5)  combining  3)  and  4),  since  no  overlappings  are 
allowed,  we  we  must  conclude  (Y,K)  =  (3,4)  and  (M,W)  =  (8,6);  6)  since 
7  la  the  only  remaining  number,  "A"  must  be  7  .  Note  that  we 
could  reverse  1)  2)  3)  to  3)  2)  1)  ,  since  3)  can  be  reached 
Independently  and  it  gives  a  sufficient  information  to  discover  2)  . 

Also  4)  does  not  need  both  1)  and  3)  ,  since  we  can  reach  it  directly 
from  2)  .  For  these  reasons,  strictly  speaking,  there  are  some  minor 
variations  in  this  strategy,  and  the  above  problem  is  not  exactly  an 
example  of  the  single  strategy  case.  In  spite  of  those  variations, 
however,  our  observation  tells  us  that  the  majority  of  subjects  take  the 
above  strategy,  probably  because  1)  is  easier  to  discover  than  3),  and 
so  forth.  Table  6-1  presents  the  attainment  scores  and  the  description  of 
the  corresponding  subprocess  for  each  score  in  this  example. 

It  has  been  reported  (Samejlma,  1969)  that,  in  their  effort  of 
solving  this  problem  in  the  paper-and-pencll  testing  situation,  as  many 
as  187  subjects  out  of  the  total  of  883  junior  high  school  students 


Y  A  M  A 
+  U  M  I 


(0346789) 
A  I  K  M  U  W  Y 


K  A  W  A 


FIGURE  6-1 

Item  Dy  y  of  LIS  Measurement  Scale  of  Non- 
Verbal  Ability. 


TABLE  6-1 

Subprocesses  of  the  Task  Involved  in  Solving  Item  D2  2 
LIS  Measurement  Scale  of  Non-Verbal  Ability. 


'  — .  "  1 

Item  Score 

"g 

Attainment 

Score  y„ 

9 

Subprocess 

0 

0 

(none) 

0 

1 

I  =  0  Left:  (346789) 

1 

2 

U  =  9  Left:  (34678) 

1 

3 

K  =  Y  +  1 

(Y.K)  =  (3,4)  or  (6,7)  or  (7,8) 

1 

4 

W  =  2M  -  10 

(M,W)  =  (7,4)  or  (8,6) 

1 

5 

M  =  8 

W  =  6 

Y  =  3 

K  =  4  Left:  (7) 

2 

6 

A  =  7 

-37- 


dlscovered  both  1=0  and  U  ■  9  and  wrote  them  down  on  their  sheets, 
although  they  did  not  reach  the  complete  solution  of  the  problem.  The 
number  of  those  subjects  Is  almost  as  large  as  that  of  those  who  reached 
the  complete  answer,  l.e.,  202  .  For  this  reason,  using  the  Item  score 
X  (*  0,1,2)  ,  which  Is  shown  In  the  first  column  of  Table  6-1,  the 

O 

operating  characteristic  has  been  estimated  for  each  Item  score  using  the 

normal  ogive  model  on  the  graded  response  level.  The  resultant  three 

operating  characteristics,  P  (6)  for  x  ■  0,1,2  ,  have  been  reported, 

*g  * 

and  are  reproduced  as  Appendix.  It  has  also  been  reported  that  this 
result  provides  us  with  a  good  Internal  consistency  which  supports  the 
validity  of  the  model. 

VII  Discussion  and  Conclusions 

Using  digraphs  and  trees,  an  attempt  was  made  to  give  a  rationale 
behind  a  general  model  for  differential  strategies,  and  some  concrete 
models  In  the  homogeneous  case  were  proposed.  The  single  strategy  case 
was  observed  as  a  special  case  of  the  multiple  strategy  case. 

A  question  may  arise  as  to  which  estimate  of  the  latent  trait 
should  be  taken  if  the  subject  faltered  from  one  strategy  to  another  and 
did  not  reach  the  solution  of  the  problem.  One  answer  to  this  question 
may  be  to  take  the  attainment  score  of  the  strategy  that  he  took  last, 
and  use  Its  corresponding  operating  characteristic  In  estimating  his 
latent  trait.  An  alternative  answer  may  be  to  compare  the  resultant 
estimates  of  d  obtained  by  choosing  the  different  strategies  the  subject 
has  taken  and  select  the  highest  estimate. 
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The  usefulness  of  the  model  Is  yet  to  be  discovered.  The  example 
of  the  Item  ^2,2  Measurement  Scale  provides  us,  however, 

with  some  support  for  considering  cognitive  processes  In  the  framework  of 
latent  trait  theory. 

One  of  the  difficulties  In  any  quantification  of  cognitive  data  may 
be  their  small  sample  sizes.  It  will  be  useful  to  conduct  Intensive, 
observant  laboratory  studies  as  a  pilot  study,  and  then  to  use  their 
results  In  designing  more  extensive  studies,  like  paper-and-pencll  testing, 
computerized  adaptive  testing,  and  so  forth,  for  the  purpose  of  estimating 
the  operating  characteristics  and  eventually  the  subject's  latent  trait. 
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